Fast Design Space Exploration of Nonlinear Systems: Part II

Nonlinear system design is often a multi-objective optimization problem involving search for a design that satisfies a number of predefined constraints. The design space is typically very large since it includes all possible system architectures with different combinations of components composing each architecture. In this article, we address nonlinear system design space exploration through a two-step approach encapsulated in a framework called Fast Design Space Exploration of Nonlinear Systems (ASSENT). In the first step, we use a genetic algorithm to search for system architectures that allow discrete choices for component values or else only component values for a fixed architecture. This step yields a coarse design since the system may or may not meet the target specifications. In the second step, we use an inverse design to search over a continuous space and fine-tune the component values with the goal of improving the value of the objective function. We use a neural network to model the system response. The neural network is converted into a mixed-integer linear program for active learning to sample component values efficiently. We illustrate the efficacy of ASSENT on problems ranging from nonlinear system design to design of electrical circuits. Experimental results show that ASSENT achieves the same or better value of the objective function compared to various other optimization techniques for nonlinear system design by up to 54%. We improve sample efficiency by 6-10x compared to reinforcement learning based synthesis of electrical circuits.Comment: 14 pages, 24 figures. arXiv admin note: substantial text overlap with arXiv:2009.1021

AI framework for improved system design and explainable decisions

System design space exploration requires searching for inputs that help achieve the desired performance. One popular technique used to search a large space is to sweep over several possible values of the inputs and select the combination that attains the best performance. While randomly searching for different choices for the values of the components may work for a small set of inputs, the search space explodes combinatorially as the number of inputs increases. Moreover, when the system evaluation is expensive, the designer needs to minimize the number of system evaluation calls. If the system has multiple performance metrics, the designer needs to identify the set of inputs that achieves the best tradeoff among competing performance metrics and select the input corresponding to the desired tradeoff. Real-word system design often requires designing the same system multiple times with varying specifications to meet different customer needs. For example, when designing an electrical circuit, the requirements for acceptable noise and the gain of the circuit may differ depending on the use case. In such scenarios, the designer may wish to dynamically drive a system from one use case to another without starting the search from scratch. Piggybacking on an existing solution to cater to different specifications may help reduce the design time while leveraging past knowledge. Furthermore, when the system evaluation is expensive, the designer may wish to gauge how much confidence to place in the solution suggested by the design tool before performing an evaluation. Having a confidence measure allows the designer to assess whether to evaluate a lower confidence solution with the possibility of getting a larger improvement in performance or vice versa. In addition, several systems have a multi-modal behavior: same performance over multiple choices of inputs. However, the cost of using different combinations of inputs may vary, even though the system attains the same performance. In such scenarios, the design tool should identify all combinations of possible inputs corresponding to the same performance so that the designer may choose one combination over another. When designing a complex system, the designer may often be limited by the number of inputs that can be varied. For instance, the designer may only have the flexibility to vary three out of 10 system inputs. Such limitations may arise when the system requires evaluation across different domains, and the designer wishes to vary the inputs corresponding to the domain with fast evaluation period. However, the designer may still like to characterize the performance across all the domains. In such scenarios, the design tool should have the flexibility to specify the fixed system inputs and obtain values of the remaining inputs. Finally, real-world data often have errors where some of the values of the features may be corrupted. Using erroneous data to make decisions may have catastrophic consequences. A popular technique to ensure that downstream tasks do not use incorrect data for decision-making involves identifying if the observation does not lie within the distribution of the past data (without errors). However, such a methodology ignores the valuable information that resides in features with correct values. In such scenarios, a tool that detects errors and locates/corrects the values of the features with errors will be beneficial. In this thesis, we address the system challenges mentioned above. First, we introduce a sample-efficient system design framework called ASSENT. ASSENT uses a two-step methodology for system design. In Step 1, we use a genetic algorithm (GA) to discover the rough tradeoff between performance metrics. As GA is sample-inefficient, we terminate it prematurely, thus avoiding a large number of simulations required to discover the entire tradeoff curve of different performance metrics. The designer then selects a solution of interest from the tradeoff curve to enhance the performance of the selected solution further. In Step 2, we convert a neural network verifier into an optimizer to improve the performance of the selected solution. We use an inverse design methodology that specifies the desired system performance to perform targeted simulations and drive the selected solution to perform better on all metrics. The targeted simulations, with the use of inverse design, makes the design process sample-efficient. Next, we present a framework, called INFORM, which performs constrained multi-objective optimization for system design. We introduce three inverse design techniques based on a neural network verifier, a neural network, and a Gaussian mixture model (GMM). The combinations of these inverse design methods lead to a total of seven inverse design schemes. Similar to ASSENT, we use a two-step methodology. In the first step, we modify a GA to make the design process sample-efficient. We inject candidate solutions into the GA population using inverse design methods instead of determining the candidate solutions for the next generation using only crossover and mutation, as in standard GA. The candidate solutions for the next generation are thus a mix of those generated using crossover/mutation and solutions generated using inverse design. At the end of the first step, we obtain a set of non-dominated solutions. In the second step, we choose a region of interest around the non-dominated solutions or another reference solution to further improve the objective function values using inverse design methods. Finally, we present REPAIRS: an explainable decision framework to complete/optimize partially specified systems and detect/locate/correct errors in a data instance. We use a GMM to learn the joint distribution of the system inputs and the corresponding output response (objectives/constraints). We use the learned model to complete a partially-specified system where only a subset of the component values and/or the system response is specified. When the system response exhibits multiple modes (i.e., the same response for different combinations of input values), REPAIRS determines the combinations of input values that correspond to the several modes. We also use REPAIRS for verifying the integrity of a given data instance. When the integrity check fails, we provide a mechanism to identify and correct the error

INFORM: Inverse Design Methodology for Constrained Multi-objective Optimization

Many system design methods use population-based optimization or a surrogate model for solving constrained multi-objective optimization. When designing a system with multiple objectives and constraints, the designer may first be interested in understanding the trade-offs among different objectives from a small number of simulations. In the next step, the designer may focus on specific regions of interest in the design space near a set of non-dominated solutions to further improve performance on the targeted objectives. This may help make the search process sample-efficient. We propose INFORM: a two-step approach for sample-efficient constrained multi-objective optimization of real-world nonlinear systems. In the first step, we modify a genetic algorithm (GA) to make the design process sample-efficient. We inject candidate solutions into the GA population using inverse design methods instead of determining the candidate solutions for the next generation using only crossover and mutation, as is done in standard GA. We present three types of inverse design techniques based on a (i) neural network verifier, (ii) neural network, and (iii) Gaussian mixture model. The candidate solutions for the next generation are thus a mix of those generated using crossover/mutation and solutions generated using inverse design. At the end of the first step, we obtain a set of non-dominated solutions. In the second step, we choose the regions of interest around the non-dominated solutions to further improve the objective function values using inverse design methods. We demonstrate the efficacy of INFORM through synthesis of nonlinear systems and analog circuits. The experimental results show that INFORM reduces synthesis time by up to 29× and improves the value of the objective function by up to 33% compared to a state-of-the-art baseline design methodology

TUTOR: Training Neural Networks Using Decision Rules as Model Priors

The human brain has the ability to carry out new tasks with limited experience. It utilizes prior learning experiences to adapt the solution strategy to new domains. On the other hand, deep neural networks (DNNs) generally need large amounts of data and computational resources for training. However, this requirement is not met in many settings. To address these challenges, we propose the TUTOR DNN synthesis framework. TUTOR targets tabular datasets. It synthesizes accurate DNN models with limited available data and reduced memory/computational requirements. It consists of three sequential steps. The first step involves generation, verification, and labeling of synthetic data. The synthetic data generation module targets both the categorical and continuous features. TUTOR generates the synthetic data from the same probability distribution as the real data. It then verifies the integrity of the generated synthetic data using a semantic integrity classifier module. It labels the synthetic data based on a set of rules extracted from the real dataset. Next, TUTOR uses two training schemes that combine synthetic and training data to learn the parameters of the DNN model. These two schemes focus on two different ways in which synthetic data can be used to derive a prior on the model parameters and, hence, provide a better DNN initialization for training with real data. In the third step, TUTOR employs a grow-and-prune synthesis paradigm to learn both the weights and the architecture of the DNN to reduce model size while ensuring its accuracy. We evaluate the performance of TUTOR on nine datasets of various sizes. We show that in comparison to fully-connected DNNs, TUTOR, on an average, reduces the need for data by 5.9× (geometric mean), improves accuracy by 3.4%, and reduces the number of parameters (floating-point operations) by 4.7× (4.3×) (geometric mean). Thus, TUTOR enables a less data-hungry, more accurate, and more compact DNN synthesis